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FINITE LOCAL ENHANCEMENT 
OF COLOR VIDEO SIGNALS 




J 



This invention relates to video signal processing and more particularly to 
processing color video signals so as to enhance the video images produced by 
such signals. 

Background Of The Invention 

The design of picture reproduction systems Involves consideration of a 
number of factors which affect how a picture is perceived by the human eye, 
including but not limited to the following factors: (1) sharpness or pictorial clarity, 
(2) contrast between light areas, dark areas and the related background 
illumination, and (3) realistic color values. In this context it should be noted that 
the eye does not respond equally to radiated energy of all visible wavelengths, 
there is a wide variation in response between observers, and the eye's response 
is a function of light intensity. See W. C. Hughes, "Electronic Engineers' 
Handbook", 1st Ed., pp. 20-3, 20-4, McGraw-Hill 1975. 

Color video (television) standards are based on the manner in which the 
eye perceives colored light. In color video systems a wide range of colors can 
be reproduced for reception by the eye by the addition of only three 
monochromatic light sources, red, green and blue. The color signals produced 
by a video camera are encoded before being transmitted to a television receiver 
or other video display device, the encoded signals comprising a luminance 
information signal and color signals. As explained by Hughes (supra), p.20-3, 
the luminance of a surface is the effect on the average eye of the light emitted by 
a unit area of the surface. Luminance is the integrated effect of the eye 
response and the visible light power radiated by the surface, both of which are 
functions of wavelength. 
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Spatial frequency is the frequency at which an image changes over 
distance. Luminance is more significant than color at higher spatial frequencies. 
The human eye response to high spatial frequency information is not the same 
as it is for low spatial frequency information. Accordingly efforts have been 
.made to process spatial frequency video information signals for the purpose of 
enhancing the sharpness of reproduced video images. One such effort is 
represented by U.S. Patent No. 4,721,998, issued 1/26/88 to W. E. Glenn for 
"Method And Apparatus For Processing Color Video Signals". Other such efforts 
may be known to persons skilled in the art. 

The idea of enhancing the images produced by a video camera is 
particularly intriguing in the case of producing images for surgical or medical 
purposes. For such purposes, video cameras are utilized in the form of video 
endoscopes, so-called because the camera is embodied as an integral part of 
the endoscope, or as an accessory that is attached to the back end of a 
conventional optical endoscope. In many surgical procedures it is often helpful 
to enhance color video images in a way which will add to the information 
perceived by the medical specialist. Details such as small or large blood 
vessels, differences in tissue condition or tissue types, and the margins between 
them can be made more distinct and informative by appropriately electronically 
enhancing the image reproduced by use of the video signals. 

Objects And Summary Of The Invention 

A primary object of the invention is to provide a novel method of 
processing video signals representative of an image so as to produce an 
enhanced reproduction of that image, or a portion thereof. 

Still another object is to provide a method and apparatus for electronically 
processing video signals representing images of an object or scene so that 
details of some portion or all of said object or scene are enhanced in the video 
display reproduction of said images. 



KYU-03 




These and other objects are achieved by deriving one or more digital 
color-representative signals and a digital luminance signal from a suitable 
source, for example, a color video camera or a video recording system or from 
previously processed video signals. The digital luminance signal is passed 
through a low-pass FIR (finite impulse response) filter that removes the higher 
spatial frequency components. The resulting low pass digital data is subtracted 
from the original luminance image data, leaving the digital data representing the 
high-pass image components that were rejected by the low-pass filter. The high- 
pass digital data for each pixel is subjected to amplification ("boost") according to 
a function of the low-pass data value at that pixel. Essentially the high 
frequency components of different pixels in an image are boosted differently 
according to the low spatial frequency components of the same pixels. The 
lower the value of the lower spatial frequency components, the more the higher 
spatial frequency components get boosted. The low-pass data and the 
selectively boosted high-pass data are subsequently added together to form the 
finite local enhancement version of the luminance data. The finite local 
enhanced luminance data is used to generate a display of the observed image 
with all or part of the displayed image being enhanced. To maximize the benefit 
of the enhancement process, at each pixel the enhanced values of luminance 
are compared against predetermine minimum and maximum values allowed for 
image data. Values beyond the valid range are set equal to the predetermined 
minimum or maximum values respectively. 

The invention enables different degrees of enhancement of the light and 
dark areas of the image, with the darker parts of the picture getting more boost 
than the lighter areas, unless the operator deliberately selects some other result. 
The operator may interface with the boost table by selecting a region of the 
image to be enhanced, and/or by selecting a profile of boost values (less and/or 
greater than unity) to produce a desired enhancement result. 
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Other features and advantages of the invention are described or rendered 
obvious by the following text and the accompanying drawings. 



The Drawings 
: ig. Vis a 



Fig. Vis a block diagram illustrating a video signal processing system 
embodying the present invention; 

Fig. 2^a block diagram that generally illustrates how the color video signals 
from a camera or other sources are processed to generate a luminance signal 
and a chrominance signal; 

Fig. 3-ns a block diagram illustrating a preferred embodiment of the 
luminance-enhancing system of the present invention; 

Figs. 4A-4^apB schematic examples of eight different gain tables; and 
Figs. sX^Eare schematic examples of different regional control tables. 
In the drawings, like components are identified by like numerals. 

Description of Preferred Embodiment of the Invention 
P-N>Referring to Fig. 1 , there is shown a video/apparatus comprising a luminance- 
enhancing digital video signal processing system 4 that embodies the present 
invention. The processing system 4 is adapted to receive and process video 
signal inputs derived from a suitable soi/rce, e.g., by RF transmission from a 
remote transmitter, a video recorder, or a video camera. In Fig. 1 the source of 
the color video signal input is a colo/television camera 2. By way of example 
but not limitation, the camera may be a stand alone conventional color television 
camera or a video camera as incorporated in or coupled to an endoscope or 
other apparatus. Endoscopes incorporating video cameras are well known and 
are exemplified by the devices/disclosed in U.S. Patents Nos. 4,745, 471 issued 
to K. Takamura et al.; 4,868,645 issued to K. Kobayashi; 5,577,091 issued to N. 
Akui et al.; and 5,891,015, issued to F. Strahle. It is to be understood that if the 
multi-component electrical/output signal from the camera is in analog form 
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(which currently is the most^ommon format), it is necessary to subject it to 
analog-to-digital conversion! by means of an AID converter 3 before applying it to 
the signal processing system 4. 

The digital output of signal processing system 4 may be applied without" 
material change to a video display device 6 and/or a video recorder 8 and/or a 
transmitter 10 if the latter are digitally-operating devices. Alternatively, the digital 
output signal from signal processing system 4 may be subjected to digital-to- 
analog conversion via a D/A converter 5 if display 6, recorder 8 and transmitter 
10 are analog rather than digital devices. 

Referring now to Fig. 2, digital red (R), green (G), and blue (B) color 
component signals derived from camera 2 or another source are applied to a 
summing matrix 14 where they are combined to produce a digital luminance 
signal Y and a digital chrominance signal C which constitutes phase-encoded 
RGB color data. The Y and C digital signals includes timing data for keeping 
track of pixels. As is obvious to persons skilled in the art, the color component 
data output of summing matrix 14 may consist of other digital color- 
representative signals, e.g., conventional color difference signals such as R-Y 
and B-Y, or other forms of digital signals which include, in any form, color 
components data. 

Turning now to Fig. 3, the digital signal processing system 4 comprises a filter 
16, a subtracting unit or subtractor 18, horizontal and vertical counters 20 for 
tracking pixels, a regional control tables memory unit 22 containing a plurality of 
selectively accessible digital regional control tables, a regional selection control 
23, a gain tables memory storage unit 24 containing a multiple of accessible 
stored digital gain tables, a multiplier 26, a summing unit or adder 28, a limiter 
30, and a compensating delay device 32. The filter 16 is a low-pass FIR (finite 
impulse response) filter that removes higher spatial frequency components, from 
the electrical signal output Y of summing matrix 14. The storage units 22 and 
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24 may take various forms, e.g., a storage disk or one or more E-proms forming 
part of a computer. 

The regional control memory unit 22 comprises a memory unit in which are 
stored one or more region selection control tables for generating gain table 
select command signals that are used to selectively access the gain tables 
stored in storage unit 24, whereby to vary application of the gain (boost) factors 
that are applied to multiplier 26 on a pixel-to-pixel basis ahd thereby vary the 
enhancing effect of processing system 4 on selected portions of the image 
represented by luminance signal Y and chrominance signal C. The region 
selection control 23 ts adapted to respond to operator inputs to modify operation 
of signal processing system 4 by selecting specific regional tables from memory 
unit 22, so as to vary the area of the observed image that is to be enhanced 
and/or vary the degree of enhancement. 

The storage unit 24 contains in memory a plurality of different gain tables that 
provide different gain factors for different low spatial frequency values. 
Preferably also the storage unit 24 contains in memory one or more tables that 
provide gain factors less than unity 

As is obvious to a person skilled in the art, certain of the foregoing 
components may comprise portions of a digital computer. Preferably, but not 
necessarily, subtracter 18, counters 20, regional table memory unit 22, regional 
control 23, gain tables memory unit 24, multiplier 26, adder 28 and limiter 30 
comprise portions of a digital computer, as indicated by the broken line 7, with 
the operator inputs to the computer being fed in by keyboard or voice 
commands. Also two or more of the foregoing devices may be combined into 
one discrete unit or location, e.g., memory units 22 and 24 may comprise a 
single E-prom or separate storage addresses on a single hard disk. 

The chrominance signal C is passed through the compensating delay device 
32 which is adapted to delay that signal by an amount sufficient to have it arrive 
at video display device 6 and/or video recorder 8 and/or transmitter 10 in time- 
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step with the enhanced luminance signal, i.e., at the output end of signal 
processing system 4 the chrominance data for each pixel is coincident in time 
with the enhanced luminance data for that pixel passed by limiter 24, which also 
includes timing and pixel location data. 

Still referring to Fig. 3, the luminance signal Y, which also includes timing and 
pixel location data, is applied to low-pass filter 16, and also to subtracting unit 18 
and horizontal and vertical counters 20. The filter 16 removes the higher spatial 
frequency components from the luminance signal Y, producing a modified 
luminance signal Y1 that contains the low spatial frequency image data 
components of the original luminance signal Y. The low-pass digital output Y1 of 
filter 16 is applied to subtracting unit 18, where its low spatial frequency image 
data is subtracted from the luminance image data of the original luminance 
signal Y, with the result that the output of subtracting unit 18 is a digital 
luminance signal Y2 that consists of the high-pass image data components that 
were rejected by the low-pass filter. The low-pass digital output Y1 of filter 16 is 
also applied to the multiple gain tables memory unit 24 and to adder 28. 

The counters 20 track the pixels, and the output signals of those counters are 
applied to regional memory unit 22, which also is adapted to receive operator 
inputs from regional selection control 23 for modifying the selection of the 
regional control tables stored in regional table memory unit 22, whereby to vary 
the gain table select command signal fed from regional table memory unit 22 to 
gain tables memory unit 24. 

The operator inputs to region selection control 23 may be applied by 
keyboard or voice or by other suitable means used for inputting instructions to a 
digital computer. Programming such instructions for region selection control 23 
may be accomplished in various ways obvious to persons skilled in the computer 
programming art 

The operator inputs are in the form of instructions for selectively causing the 
processing system 4 to produce the following results: full screen enhancement 
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in which the entire image is subjected to the luminance enhancement process, 
regional screen enhancement in which onfy a selected region or regions of the 
image is enhanced, split screen enhancement in which the image is split into two 
side-by-side portions and only one portion i^ enhanced, adjustment of the overall 
enhancement level, adjustment of the size of the region being enhanced, 
deliberate reduction of the luminance values in the areas surrounding the area 
undergoing luminance enhancement, and effectively nulling or by-passing the 
enhancement process. Thus, for example, full screen enhancement involves 
enhancing all of the pixels of each image, while regional enhancement involves 
enhancing only selected pixels. Nulling or bypassing the enhancement process 
is achieved by programming the region selection control so that it causes the 
memory unit 24 to provide a gain factor of unity to multiplier 26.. 

The gain table select command signal output of regional control memory unit 
22 is applied to memory unit 24 and functions to (a) select which of the various 
gain tables stored in memory unit 24 is read to determine the gain to be used by 
multiplier 26 to enhance the high frequency data luminance signal Y2, and (b) 
determine which of the pixels of the image are to be enhanced according to the 
operator inputs, if any. Depending upon the operator input(s), the gain table 
select command signal output of memory unit 22 also may function to apply a 
specific gain factor to multiplier 26. Thus, for example, if an image softening 
effect is desired for a specific region, the region control 23 may be instructed by 
the operator to regional control table in memory unit 22 that will result in a gain 
table select command that selects a gain table that provides a gain less than 
unity for all values of Y1 for pixels that fall within a selected region. It is 
preferred that processing system 4 be programmed so that, in the absence of 
any operator input, its normal mode of operation is to apply the enhancement 
process to all of the pixels of the image represented by the incoming luminance 
and chrominance signals Y and C. 
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The low pass digital signal Y1 is applied to gain tables memory unit 24 so 
that, regardless of which of its gain tables is selected by the gain table select 
command signal output of memory unit 22, the gain factor represented by the 
gain signal output from memory unit 24 will vary according to the low pass data 
value of low pass signal Y1 . 

The digital high pass data signal output Y2 of subtracting unit 18 is applied to 
multiplier 26 where it is amplified according to the gain (boost) factor applied to 
the multiplier from the multiple gain memory unit 24. The result is a boosted high 
pass luminance signal Y3. The latter is summed with the low pass luminance 
n signal Y1 in adder 28 so as to produce an enhanced luminance signal Y4. That 

y3 enhanced luminance signal Y4 may be applied without limitation to the video 

P display device 6 and/or video recorder 8 and/or transmitter 1 0 in time-step with 

% the chrominance signal C. However, in order to maximize the benefit of the 

O enhancement process, it is preferred to pass the signal Y4 through limiter 30 

z'Ti 

J" which is adapted to compare the enhanced luminance values with predetermined 

Q maximum and minimum values and to (1 ) change those values that are below 

fy the predetermined minimum value so that they equal that minimum value, and 

(2) change those values that exceed the predetermined maximum value so that 
' they equal the maximum value. 

To summarize the operation of processing system 4, each pixel of high-pass 
image data represented by high pass data luminance signal Y2 is subjected to 
amplification ("boost") according to a gain factor provided by the gain tables in 
storage unit 24 that varies as a function of the low-pass data value at that pixel 
represented by the low-pass luminance signal Y1. This enables different 
degrees of enhancement of the light and dark areas of the image. The operator 
may interface with the gain table by selecting, via the signal output of region 
selection control 22, a specific region of the image to be enhanced and a profile 
of boost values which produce the enhancement desired. The signal Y4 
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represents the finite local enhancement version of the luminance data contained 
in the original luminance signal Y. 

Figs, 4A-4H provide examples of eight different gain tables stored in gain 
table memory unit2H. For convenience the gain tables in Figs. 4A-4H are 
numbered from 0 to 7 so as to distinguish one from the other. The first three 
gain tables, tables 0, 1 and 2 (Figs. 4A-4C), provide no enhancement, while the 
other five tables, tables 3 to 7 (Figs. 4D-4H), provide five different enhancement 
levels. Fig. 4A illustrates a gain table for applying a gain of 0 (zero) to multiplier 
26 for all low pass frequency values (from black to white). This table is used for 
regions in which the high frequency values are to be eliminated completely, 
y3 thereby softening the image as much as possible. Fig. 4B shows a gain table in 

5 which a gain of 0.5 is passed to the multiplier in response to all low pass 

4jj frequency values. This gain table is used where it is desired to soften a region 

0 from its normal appearance partially. Fig. 4C differs in that a gain of 1 is 

^ provided in response to all low pass frequency values. This gain table is used 

Q where it is desired to show pixels with no modification of their normal luminance 

fij value. Fig. 4D represents the first of five different enhancement levels. In the 

1 ^ case of Fig. 4D, the gain non-uniformly varies from 1 .3 for low frequency values 
m representative of black image areas to 1 .0 for low frequency values 

representative of white image areas. The table of Fig. 4E represents a second 
enhancement level. It varies the gain in equal decrements from 1 .7 for low 
frequency values representative of black image areas to 1 .0 for low frequency 
values representative of white image areas. Fig. 4F shows a gain table for a 
third enhancement level. In this case the low frequency values representative of 
black image areas are provided with a gain of 2.7, with the gain factors changing 
in a non-uniform manner to a low of 1 .0 for low frequency values representative 
of white image areas Fig. 4G provides a gain table for a fourth enhancement 
level, wherein the gain factors vary non-uniformly from 3.5 for black image areas 
to 1 .0 for white image areas. Fig. 4H provides a fifth enhancement level, 



KYU-03 



H 



11 



wherein the gain factors vary non-uniformly from a maximum value of 4.0 for 
black image areas to a value of 1 .0 for white image areas. 

Figs. 5A-5E provide schematic illustrations of selected examples of regional 
control tables stored in regional control table memory unit 22. In each figure, 
the table is represented as consisting of 9 squares on each side, with each 
square representing a pixel and pixel location. Additionally each pixel has a 
number representing one of the eight gain tables of Figs. 4A-4H. 

It is to be understood that the number of pixels that are processed by the 
invention will vary according to the number of pixels making up the image- 
receiving surface of the electronic imaging device (CCD array or like solid state 
device) that is used to generate electrical output signals representative of optical 
images focused on that surface. Accordingly it should be understood that in 
practice each regional control table will consist of data for a substantially greater 
number of pixels than are represented in Figs. 5A-5E. 

In Fig. 5A, a small rectangular center region consisting of 9 squares, each 
identified by the numeral "7" to indicate the gain table of Fig. 4H), is surrounded 
by other squares each identified by the numeral "0" to indicate the gain table of 
Fig. 4A. Accordingly the pixels assigned the numeral "0" will the maximum 
softening effect since the table of Fig. 4A has a uniform gain of zero (0.0), while 
the pixels identified by the numeral "7" will be subjected to the gain factors of the 
^ gain table of Fig. 4H$. In Fig. 5B, the enhanced region is enlarged to consist of 
25 center located pixels, with the remaining surrounding pixels being softened 
due to being subjected to zero gain according to the table of Fig. 4A. Fig. 5C 
also has a small rectangular center region consisting of 9 pixels each identified 
by the numeral "7" to indicate that they will be enhanced according to the gain 
table of Fig. 4H, but in this case the immediately adjacent pixels on all four sides 
of the rectangular center region are identified by the numeral "2" indicative of the 
gain table of Fig. 4C, and the immediate next rows of pixels are identified by the 
numeral "1" indicative of the gain table of Fig. 5B, while the outermost pixels are 
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identified by the numerical value of "0" indicative of the gain table of Fig. 4A. 
The net effect is that the pixels in the center region will be enhanced by 
subjecting them to a gain factor as determined by the table of Fig. 4H, and that 
enhanced center region of the image is surrounded by pixels that have a 
graduated "softer" regions that subjected to gain factors determined by the gain 
tables of Figs. 4C, 4B and 4A. Fig. 5D illustrates a split screen, wherein five 
columns of pixels are enhanced by subjecting them to gain factors according to 
the gain table of Fig. 4H, and the remaining four rows are characterized by no 
enhancement in luminance due to being subjected to the gain factor of unity 
according to the gain table of Fig. 4C ("enhancement level 2). In Fig. 5E all of 
the pixels identified by the numeral "7", indicating that they are all subjected to 
enhancement according to the gain table of Fig. 4H. 

It is to be understood that other regional control tables may be stored in 
memory unit 22 in addition to or in place of the regional control tables shown in 
Figs. 5A-5e. Also other gain tables may be incorporated in addition to or in 
place of the gain tables shown in Figs. 4A-4H. The system may be programmed 
so that the selection of particular regional control tables is automatic. 
Alternatively, and preferably, the system is programmed so that it automatically 
accesses a regional table is like that shown in Figs. 5A-5C, so that the center 
portion of an image is enhanced and the surrounding regions are not enhanced 
or in fact subject to a gain less than unity. Predetermined selection of other 
region control tables may be programmed into the system for particular 
applications. 

The invention provides a luminance enhancement system that is reliable and 
provides a wide selection of image enhancement formats. Also the system is 
compatible with and may be embodied in existing imaging systems. It has been 
found to be particularly advantageous in enhancing images produced by a video 
endoscope so as to make it easier to discern details and differences in live 
tissue. In this respect, use of a regional control table patterned like those 
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represented in Figs. 5B and 5C has been shown to greatly increase the ability of 
a surgeon to discriminate between different areas of tissue being inspected by 
means of a video endoscope. A further advantage of the invention is that the 
enhanced luminance and colorance signals may be readily converted to analog 
formats for use in operating analog-type display, video recording or video 
transmitting apparatus. Alternatively, those digital data signals may be used to 
operate digital display, recording or transmitting apparatus. 

It is recognized that persons skilled in the art may readily discern that the 
invention is susceptible to a number of changes, and in particular that the 
substantially entire processing system 4 may be embodied in or take the form of 
a digital computer. Still other changes will be obvious to persons skilled in the 
art from the foregoing specification. Accordingly the invention shall be deemed 
to be defined by the following claims which are to be construed in light of the 
foregoing specification 
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